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ABSTRACT 



The process of assessment in career and technical education 
(CTE) is changing significantly under the influence of forces such as 
emphasis on assessment for individual and program accountability; emphasis on 
the investigation of consequences of assessment; emergence of item response 
theory, which supports computer adaptive testing; and pressure for authentic 
assessment by critics of standardized testing. The use of technology for 
assessment is being increasingly stressed. Potential advantages of 
computerized assessment include rapid feedback, money savings, enhanced 
security, more curriculum time, and capability to track process-oriented 
variables. Potential disadvantages include costs of equipment, personnel, and 
training; increased marginalization of groups based on ethnic or 
socioeconomic status; and missed opportunities to implement authentic 
assessments. Examples of applications of technology to evaluation in CTE 
include test design, item creation, presentation, item scoring, and location 
(through use of the Internet) . In moving toward computerized assessment, 
consequences should be considered in line with current evaluation and 
validation models. Especially for policy makers and administrators, 
implementation is an important aspect of computerized assessment. Technology 
can support the broad changes in assessment that are ongoing as a function of 
internal and external scrutiny, and both authentic as well as traditional 
assessment can benefit from technology. (Contains 24 references.) (KC) 
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Technology and Assessment 



Educational technology has ranged from 
blackboards to audiovisual media, but 
computers, networks, and hypermedia 
comprise a new frontier. Technology ap- 
plies in principle to curriculum, instruc- 
tion, and assessment. Within this matrix, 
technology is a springboard from which 
career-technical assessment can leap into 
the future. We discuss the evolution of as- 
sessment and then present computerized 
assessments and simulations. We conclude 
with implementation issues for career- 
technical education (CTE), including ac- 
cess by multiple populations. 



Assessment Is Evolving 



Assessment is changing significantly un- 
der the influence of forces that are social 
(National Research Council 1999) and 
technical (Hambleton 1996). Major forces 
include — 

• Assessment for individual and program 
accountability, often high- stakes in na- 
ture 

• An emphasis on the investigation of 
consequences of assessment in the re- 
vised Standards for Educational and 
Psychological Testing (AERA, APA, 
NCME 1999). 

• Emergence of item response theory 
(IRT) , which supports computer adap- 
tive testing (CAT) and creates “new 
rules” of measurement (Wainer 2000). 

• Pressure for authentic assessment by 
critics of standardized testing (Kohn 
2000; Wiggins 1993) 

Wiggins (1993) asserts that appropriate 
assessments are valid, authentic, and con- 
textual. He argues that context is crucial 
for test designers because competence is 
situational and its essence is analysis of the 
situation and its shifts. His profile aids in 
developing assessments that are authen- 
tic and suggests roles for technology in as- 
sessment. A hypothetical example is a 
simulation for automotive technicians that 
begins with a customer complaint and pro- 
ceeds through diagnosis, repair, and test- 
ing. In this example, instruction and test- 
ing can be fused seamlessly, as done in 
embedded assessment systems for science 
education (Wilson and Sloane 2000). 

o 
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Technology Applications 



In general, discussions of technology origi- 
nate outside of CTE, and discussions of 
assessment and instruction are often sepa- 
rated. Bunderson, Inouye, and Olsen 
(1989) identified four generations: com- 
puterized testing (CT), computer adaptive 
testing (CAT), continuous measurement 
(CM), and intelligent measurement (IM). 
CT is the computer as a page-turner and 
scorer, CAT adjusts items to the test-taker, 
CM embeds assessments into ongoing 
learning, and IM adds expert knowledge 
that enhances testing and interpretation. 
Assessment via computer can be central- 
ized or distributed over a network. What 
are the pros and cons? 

Potential advantages of computerized as- 
sessment are — 

• rapidity of feedback provided to learn- 
ers and to teachers, 

• monetary savings over time due to re- 
duction in printing and shipping costs, 

• enhanced security of summative high- 
stakes assessments, 

• more curriculum time because of time 
savings in assessment, 

• capability to track process-oriented 
variables (time, sequence). 

Potential disadvantages include — 

• costs including equipment upgrading, 
personnel, and professional develop- 
ment, 

• increased marginalization of groups 
based on demographic or socioeco- 
nomic status, [e.g., the “digital dilem- 
ma” (Van Dusen 2000)], 

• missed opportunities to implement 
authentic assessments meaningfully. 

For Bennett (1999), technology influences 
assessment in many ways. Although he did 
not emphasize CTE, examples of applica- 
tions include — 

• Test design. A system, Portal, that em- 
ploys “models” of student, evidence, and 
task to organize the domain. Application 
could be accomplished by ensuring that 
these three types of models reflect CTE 
students, national skill standards, and re- 
alistic entry-level tasks. 



• Item creation. A Mathematics Test Cre - 
ation Assistant program generates items for 
human review based on specification of a 
few variables. Application to CTE might 
consist of providing additional context that 
is specific to the 16 Department of Educa- 
tion career clusters. 

• Presentation. Simulations that “game” 
jobs or parts of jobs. Consider the Interac- 
tive Patient developed at Marshall Univer- 
sity as a case management exercise. In CTE 
this simulation could be adapted to diver- 
sified health occupations (or others) that 
feature interactions aimed at diagnosis, or 
the WonderTech toy factory could be used. 

• Item scoring. Here the focus is on scor- 
ing constructed rather than chosen re- 
sponses (essays instead of the multiple 
choices of traditional tests). Application 
to CTE could include responses involved 
in general entrepreneurship aspects of any 
occupation-specific training (e.g., market 
analysis, site selection, business plan prepa- 
ration). 

• Location. An analogy is made between 
distance learning (instruction) and Inter- 
net testing (assessment). For CTE, this 
analogy suggests the power of distributing 
assessment (and instruction) via Internet. 

The Office of Technology Assessment 
(1994) study of vocational education test- 
ing reviewed assessments of four skill sets 
(academic, vocational, generic workplace, 
and broad technical), and several vendors 
(e.g., NOCTI, ACT, V-TECS). Two con- 
clusions from the report were that perfor- 
mance assessments were more common in 
CTE and that broad technical skills were 
not being assessed well. Technology could 
assist in maintaining the former finding 
and addressing the latter problem. 

Internet testing for CTE has been investi- 
gated in two studies, both using the com- 
puter as a “page-turner” (computer pre- 
sents and scores the test). Kapes et al. 
(1998) tested 360 students in Child Care 
and Auto Body Repair programs using both 
paper-pencil and Internet formats for com- 
parison. No significant differences in test 
scores were observed across demographic 
and special needs groups. Attitude surveys 
indicated that students preferred Internet 
testing 3 to 1. The Internet was a cost-ef- 
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fective and feasible alternative to tradi- 
tional assessment, and test security was not 
compromised. Austin and Mahlman 
(2000) reported a small demonstration in- 
volving Administrative Office Technology 
students. A 100-item multiple 'choice test 
was administered via Internet to 125 stu- 
dents. A subset of these students had taken 
the test earlier in a paper- pencil format. 
The scores for Internet administration of 
that subgroup were significantly higher but 
exhibited similar variability and internal 
consistency reliability. Teachers and learn- 
ers had positive reactions to the feedback 
provided by the assessment system. What 
these two studies indicate is that even a 
relatively primitive application, the com- 
puter as a page -turner and scorer rather 
than as adaptive administrator, was effec- 
tive and was received well by stakehold- 
ers. 

Another way that technology facilitates 
authentic assessment is simulation. Simu- 
lations are linkable, for example, to the 
SCANS foundation-competency frame- 
work. A major advantage that we foresee 
is linking instruction and assessment to 
achieve the integration urged by Sheppard 
(2000). Computer simulations have high 
front-end costs of development to ensure 
realism, but their operational costs should 
be low. The issue of integration of assess- 
ment into instruction remains. 



Implementation Issues 



In moving toward computerized assess- 
ment, consequences should be considered 
in line with current evaluation and vali- 
dation models. In addition to technical is- 
sues, it is important to understand the ef- 
fects on groups that have been margin- 
alized in the past. Examples include gen- 
der, racial/ethnic, and disability status at 
the individual level, and district socioeco- 
nomic levels at the institutional level. 

Especially for policy makers and adminis- 
trators, implementation is an important 
aspect of computerized assessment. Presen- 
tation of advantages vs. disadvantages, as 
well as usable strategies, adds value to dis- 
cussions of technology. In CTE, a system 
involving modularized Internet testing is 
being developed for Information Technol- 
ogy programs. Designed to generate reli- 
able statewide accountability data, this 
system will allow instructors to register stu- 
dents to take assessment modules after 
completing instruction. Question Mark 
software supports multiple item formats 
that approach authentic assessment (al- 
though there is room for improvement). 
<F - system provides immediate feedback 
i r module attainment and also tracks stu- 



dent progress to generate program-level 
performance data for state and local ad- 
ministrators. 

For teachers, technology- aided assessment 
poses potential threats because of new 
knowledge requirements. It will be crucial 
for administrators to provide professional 
development, perhaps using distance 
learning, so that teachers feel comfortable 
about systems used for assessment. As an 
example, Waugh, Buell, and Levin (1999) 
developed the Technology Competencies 
Database to support integrated teaching 
and assessment. It stores student accom- 
plishments linked to technology compe- 
tencies, is accessible by both student and 
teacher via Internet, and supports student 
creation of portfolios based on self- submit- 
ted evidence. 

Evaluation of technology-based assessment 
should continue. Cost-benefit analyses are 
helpful in evaluating this technology. Costs 
may be developmental/one time or main- 
tenance/ongoing. Categories of benefits are 
those that accrue to student evaluation, 
program improvement, and accountability. 

Technology can support the broad changes 
in assessment that are ongoing as a func- 
tion of internal and external scrutiny. 
Authentic as well as traditional assess- 
ments benefit from technology. Although 
we noticed a far greater focus on curricu- 
lum in discussions of educational technol- 
ogy (Plomp and Ely 1996), there is a lit- 
erature on technology-aided assessment. 
This literature should be better integrated 
with practice to ensure lasting benefits for 
CTE. 



References 



American Educational Research Association, 
American Psychological Association, and Na- 
tional Council on Measurement in Education. 
Standards for Educational and Psychological 
Testing. Washington, DC: American Educa- 
tional Research Association, 1999. 

Austin, J.T., and Mahlman, R.A. “Using the In- 
ternet for Career-Technical Assessment: A 
Pilot Project in Ohio.” Journal of Career and 
Technical Education 16, no. 2 (Spring 2000): 
27-39. 

Bennett, R.E. “Using New Technology to Improve 
Assessment.” Educational Measurement : Is- 
sues and Practice 18 (1999): 5-12. 

Bunderson, V.; Inouye, D.; and Olsen, J.B. “The 
Four Generations of Computerized Educa- 
tional Measurement.” In Educational Mea- 
surement, 3d ed., edited by R.L. Linn, pp. 367- 
407. New York: Macmillan/ACE, 1989. 

Hambleton, R.K. “Advances in Assessment Mod- 
els, Methods, and Practices.” In Handbook of 
Educational Psychology, edited by D.C. Ber- 
liner and R.C. Calfee, pp. 889-925. New York: 



American Council on Education/Macmillan, 
1996. 

Kapes, J.T. et al. “Internet-based vs. Paper-pencil 
Occupational Competency Test Administra- 
tion.” Journal of Vocational Education Re- 
search 23 (1998): 201-219. 

Kohn, A. The Case against Standardized Test- 
ing. Greenwich, CE Heinemann, 2000. 

National Research Council. High Stakes: Test- 
ing for Tracking, Promotion, and Graduation. 
Washington, DC: National Academy Press, 
1999. 

Office of Technology Assessment. Testing and 
Assessment in Vocational Education. Wash- 
ington, DC: OTA, U.S. Congress, 1994. 

Plomp, T., and Ely, D.P, eds. International Ency- 
clopedia of Educational Technology, 2d ed. 
London, England: Pergamon, 1996. 

Sheppard, L.A. “The Role of Assessment in a 
Learning Culture.” Presidential Address, 
American Educational Research Association, 
New Orleans, LA, April 2000. 

Van Dusen, G.C. Digital Dilemma : Issues of 
Access, Cost, and Quality in Media-enhanced 
and Distance Education. San Francisco, CA: 
Jossey-Bass, 2000. 

Wainer, H., ed. Computer Adaptive Testing: A 
Primer. Mahwah, NJ: Erlbaum, 2000. 

Waugh, M.; Buell, J. ; and Levin, J. “The Tech- 
nology Competencies Database.” Journal of 
Technology and Teacher Education 7 (1999): 
351-363. 

Wiggins, G. Assessing Student Performance. San 
Francisco, CA: Jossey-Bass, 1993. 

Wilson, M., and Sloane, K. “From Principles to 
Practice: An Embedded Assessment System.” 
Applied Measurement in Education 13 
(2000): 181-208. 

Internet Resources 



General 

http://lrs.ed.uiuc.edu/wp/credibility/bib.html 

httpV/bubl.ac.uk/link/e/educational 

technologyresearch.htm 

http://www.jtap.ac.uk/reports/htm/jtap-028- 

7.html 

http://www.bookstoread.eom/E/ET/et.htm 

Assessment 

http://www.webmcq.com 

http://www.questionmark.com 

http://caacentre.ac.uk/ 

http://www.itworks-ohio.org 



The work reported herein was supported under the 
National Dissemination Center for Career and Tech- 
nical Education, PR/Award (No. V051A990004) as 
administered by the Office of Vocational and Adult 
Education, U.S. Department of Education. However, 
the contents do not necessarily represent the posi- 
tions or policies of the Office of Vocational and Adult 
Education or the U.S. Department of Education, and 
you should not assume endorsement by the Federal 
Government. 




National 

Dissemination 

Center 



CAREER 8 TECHNICAL EDUCATION IRE OHIO STATE UNIVERSITY 

1900 KENNY ROAD, COLUMBUS OH 43210-1090 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 

REPRODUCTION BASTS 




This document is covered by a signed “Reproduction Release 
(Blanket) form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either “Specific Document” or “Blanket”). 




EFF-089 (9/97) 




